Supplementary Material: Finding Linear Structure in Large Datasets with Scalable Canonical Correlation Analysis

1 2 ∆ φ t = φ t − φ 1 , ∆ ψ t = ψ t − ψ 1 , ∆φ t = φ t − φ 1 , ∆ψ t = ψ t − ψ 1 Further, we define cos x (u, v) = u Sxv uxvx , the cosine of the angle between two vectors induced by the inner product u, v = u S x v. Similarly, we define cos y (u, v) = u Syv uyvy. To prove the theorem, we will repeatedly use the following two lemmas. Proof of Lemma 1. The proof is in the main paper. Lemma 2. ∆φ t x ≤ 1 λ1 2 1+cosx(φ t ,φ1) ∆ φ t x and ∆ψ t y ≤ 1 λ1 2 1+cosy(ψ t ,ψ1) ∆ ψ t y Proof of Lemma 2. Notice that cos x (φ t , φ 1) = cos x (φ t , φ 1), then ∆ φ t 2 x = φ t − φ 1 2 x ≥ φ 1 2 sin 2 x (φ t , φ 1) = λ 2 1 sin 2 x (φ t , φ 1) Also notice that φ t x = φ 1 x = 1, which implies cos x (φ t , φ 1) = 1 − φ t − φ 1 2 x /2 = 1 − ∆φ t 2 x /2. Further ∆ φ t 2 x ≥ λ 2 1 sin 2 x (φ t , φ 1) = λ 2 1 (1 − cos 2 x (φ t , φ 1)) = λ 2 1 2 ∆φ t 2 x (1 + cos x (φ t , φ 1)) Square root both sides, ∆φ t x ≤ 1 λ 1 2 1 + cos x (φ t , φ 1) ∆ φ t x Similar argument will show that ∆ψ t y ≤ 1 λ 1 2 1 + cos y (ψ t , ψ 1) ∆ ψ t y 1.1. Proof of Theorem 2.1 Without loss of generality, we can always assume cos x (φ t , φ 1), cos y (ψ t , ψ 1) ≥ 0 because the canonical vectors are only identifiable up to a flip in sign and we can always choose φ 1 , ψ 1 such that the cosines are nonnegative. Apply simple algebra to the gradient …